Test-aware Combinatorial Interaction Testing

Combinatorial interaction testing (CIT) approaches systematically sample a given configuration space and select a set of configurations, in which each valid t-way option setting combination appears at least once. A battery of test cases are then executed in the selected configurations. Existing CIT approaches, however, do not provide a systematic way of handling test-specific inter-option constraints. Improper handling of such constraints, on the other hand, causes masking effects, which in turn causes testers to develop false confidence in their test processes, believing them have tested certain option setting combinations, when they in fact have not. In this work, to avoid the harmful consequences of masking effects caused by improper handling of test-specific constraints, we compute t-way test-aware covering arrays. A t-way test-aware covering array is not just a set of configurations as is the case in traditional covering arrays, but a set of configurations, each of which is associated with a set of test cases. We furthermore present a set of empirical studies conducted by using two widely-used highly-configurable software systems as our subject applications, demonstrating that test-specific constraints are likely to occur in practice and the proposed approach is a promising and effective way of handling them.